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Abstract 

There  are  many  challenges  facing  complex  system  development  in  today’s 
environments.  Systems  have  become  far  more  complex,  operating  in  a  net-centric  environment, 
with  ever-increasing  threats  to  system  security  posing  a  challenging  design  and  development 
task  for  program  managers  and  systems  engineers.  We  have  seen  an  increasing  number  of 
major  DoD  system  development  programs  experiencing  difficulties  and  failing  to  achieve  their 
intended  goals  successfully.  Reasons  for  these  difficulties  and  failures  include  both  technical 
and  programmatic  type  issues.  At  the  top  of  the  list  has  been  the  failure  to  properly  assess  the 
technical  maturity  of  complex  systems  during  system  development,  leading  to  cost  overruns, 
program  delays,  program  cancellations,  and  unacceptable  system  performance.  Recently 
introduced  corporate  or  program  portfolio  management  ideologies  supporting  system 
development  in  the  DoD  have  shown  some  promise  in  providing  a  more  dynamic  approach  to 
project  management.  Advantages  include  the  ability  to  make  dynamic  changes  to  the  mixture  of 
technology  investments  in  a  development  program  and  increased  probability  of  attaining  the 
desired  end-state  goals  at  planned  cost  and  on  schedule.  The  programs  need  to  consider 
external  technology  shifts  and  ensure  the  programs  and  their  technology  investments  stay 
ahead  of  the  critical  “S-Curve.”  The  dynamics  of  program  management,  including  effective 
decision-making,  also  play  an  important  role  in  ensuring  end-goal  success.  Missing  from 
corporate  portfolio  management  are  good  maturity  metrics  to  assess  the  system  development 
process  throughout  the  lifecycle.  This  paper  addresses  the  application  of  system  maturity 
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metrics  and  decision  theory  ideologies  to  a  portfolio  management  framework  supporting  multi¬ 
technology-based  system  development.  The  application  of  previous  research  performed  by  the 
Stevens  Institute  of  Technology  in  the  area  of  system  maturity  metrics,  including  “systems 
readiness  levels,”  will  be  leveraged  and  applied  to  existing  problem  sets — resulting  in  a  dynamic 
decision-making  process. 

Introduction 

As  we  look  at  current  lifecycle  system  development,  we  see  an  increasing  number  of 
major  Department  of  Defense  (DoD)  system  development  programs  experiencing  difficulties 
and  failing  to  achieve  their  intended  goals  successfully.  Reasons  for  these  difficulties  include 
both  technical  and  programmatic  type  issues  that  are  experienced  throughout  the  system 
development  lifecycle.  At  the  top  of  the  list  has  been  the  failure  to  properly  assess  the  technical 
maturity  of  these  complex  systems  during  system  development,  leading  to  cost  overruns, 
program  delays,  program  cancellations,  and  unacceptable  system  performance.  Evidence  of 
this  is  seen  in  the  often  cited  Government  Accountability  Office  (GAO)  report  that  reviewed  and 
analyzed  major  defense  acquisition  programs.  This  report  concluded  that  the  causes  and 
reasons  for  failure  in  major  defense  acquisition  programs  were  due  to  a  majority  of  programs 
failing  to  meet  a  TRL  7  level  before  entering  the  system  development  phase  (1999).  These 
findings  were  echoed  again  in  a  more  recent  GAO  report  that  showed  an  increase  from  the 
previous  year  in  the  number  of  programs  with  immature  technologies  still  maturing  technologies 
late  into  the  system  development  and  production  lifecycles  (2008).  It  is  troubling  that  nine  years 
after  the  original  report,  we  are  still  reporting  the  same  types  of  problems  with  these  acquisition 
programs.  The  evidence  is  overwhelming  and  shows  that  serious  attention  to  the  application  of 
lifecycle  system  maturity  metrics  is  essential  to  reversing  the  present  trend  in  major  acquisition 
program  failures.  Figure  1  below  shows  the  maturity  levels  of  critical  technologies  for  DoD 
programs. 


Figure  1.  Maturity  Levels  of  Critical  Technologies  for  DoD  Programs 
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System  Development  Challenges 

There  are  many  challenges  facing  system  development  in  today’s  fast-paced 
environments.  Systems  have  become  more  complex,  operating  in  a  net-centric  environment, 
with  ever  increasing  threats  to  system  security  posing  a  challenging  design  and  development 
task  for  program  managers  and  systems  engineers.  Complicating  this  scenario  are  the  added 
constraints  of  budget,  shorter  development  lifecycles,  and  available  experienced  workers. 
These  demands  have  further  increased  the  pressure  on  program  managers  and  systems 
engineers  to  achieve  expected  success  in  the  areas  of  technical  performance,  budget,  and 
schedule.  Further  concerns  are  the  failure  of  developers  to  make  the  necessary  decisions  to 
integrate  newer  technologies,  and  they  continue  to  invest  in  existing  technologies  that  produce 
no  added  benefits  while  the  rapidly  changing  technological  world  moves  on.  This  is  known  as 
the  “S  Curve”  effect  and  is  illustrated  in  Figure  2  below.  These  developers  face  the  risk  and 
unintended  consequences  of  becoming  irrelevant  quickly  by  not  reacting  fast  enough  to  these 
external  forces  (Christensen,  2003). 


Value 


Technology  not  scaling  to  the 
increasing  magnitude  and 
market  demand 


Enhancements  implemented; 
Users  more  comfortable  and 
skilled  in  using  the  technology 


Implementation  problems; 
Users  climbing  the 
learning  curve 


Jump  to  new 
technology  to 
remain  relevant 


Continued  investment  in 
legacy  systems  may 
lead  one  out  of  business 


Start  investing  in 
“Development”of 
new  technology  here 


r 

0 

U 

Investment  or  Time 

We  need  to  the  plan  to  jump  to  the  next  technology  curve  to  avoid  diminishing  returns  on 
our  investments  and  to  sustain  our  market  value  (industry)  or  our  relevance  (Government) 


Figure  2.  Technology  S-Curve 

Need  for  an  Integrated  Environment 

For  success  in  today’s  accelerated,  system  acquisition  development  programs,  we  need 
to  ensure  the  existence  of  an  integrated  environment  that  consists  of  a  management  process 
that  is  guided  by  a  defined  lifecycle  framework  and  at  the  same  time,  a  maturity  metric  process 
that  maps  to  this  same  lifecycle  framework  and  supports  the  management  process.  This 
integrated  environment  allows  for  maximum  interaction  between  these  domains  to  support  the 
manager’s  decision-making  process,  whether  the  organization  is  small,  medium,  or  large.  This 
integrated  environment  will  consists  of  the  following  three  components:  a  defined  accepted 
lifecycle  framework,  a  realistic  portfolio  management  process,  and  metrics  to  include  financial, 
technical,  and  technology  maturation.  Since  this  paper  is  looking  at  DoD  based  programs,  we 
will  refer  to  the  DoD  5000.2  lifecycle  framework.  For  the  system  maturity  metrics,  we  can  apply 
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the  System  Readiness  Level  (SRL)  model,  developed  by  Stevens  Institute  of  Technology,  to  a 
portfolio  management  based  environment,  which  is  becoming  more  popular  in  DoD  programs. 


Integrated  Environment  ^ 


V 


Figure  3.  Integrated  Environment 


What  is  a  Lifecycle  Framework? 

A  lifecycle  is  an  inherent  part  of  all  system  development  and  encompasses  a  framework 
that  that  defines  all  the  necessary  systems  engineering  phases  and  lifecycle  activities  that  are 
necessary  to  support  system  development  production  and  post  development  activities.  Within 
the  lifecycle  are  decision  points  or  milestones  when  technology,  performance,  and  schedule  are 
assessed  (INCOSE,  2006).  In  its  simplest  definition,  a  lifecycle  is  described  as  “The  system  or 
product  evolution  beginning  with  the  identification  of  a  perceived  customer  need,  addressing 
development,  test,  manufacturing,  operation,  support,  and  training  activities,  continuing  through 
various  upgrades  or  evolutions,  until  the  product  and  its  related  processes  are  disposed  of” 
(Kossiakoff  &  Sweet,  2003).  Obvious  in  Kossiokoff  and  Sweet’s  definition  is  the  existence  of 
least  three  stages,  the  conceptual  development,  engineering  development,  and  post 
development.  Within  each  stage  are  the  activities  described  in  Kossiakoff  and  Sweet’s  lifecycle 
definition.  In  the  real  world,  there  are  some  subtle  variations  in  the  comparison  of  lifecycle 
models  across  the  different  system  development  domains.  This  paper  will  focus  on  the  DoD’s 
“DoD  5000  Acquisition  Lifecycle  Framework”  model,  which  has  benefited  DoD  acquisition  based 
programs  successfully  by  provided  a  basic  common  system  development  lifecycle  framework 
describing  all  the  necessary  processes  and  activities  needed  to  support  system  acquisition. 
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Figure  4.  Lifecycle  Model  Comparisons 


What  are  Maturity  Metrics? 

In  the  past,  we  have  made  considerable  improvements  in  the  tracking  and  monitoring  of 
program  metrics  focusing  on  the  financial  status  through  improved  software  IT  systems.  We 
have  also  done  well  in  metrics  associated  with  performance  testing  of  systems.  Missing  is  the 
lack  of  better  metrics  supporting  support  the  lifecycle  assessment  of  system  maturity. 
Technology  maturity  is  a  main  area  of  concern  among  developers  as  many  system  development 
efforts  have  failed  because  of  the  inability  to  assess  the  system  technology’s  state  of  progress 
or  development.  This  can  often  lead  to  failure  of  a  technology  to  perform  in  a  system  or  be 
integrated  into  a  system.  The  need  to  assess  the  maturity  level  of  the  technologies  and  systems 
in  the  development  process  becomes  a  critical  factor  in  the  decision-making  process  throughout 
the  system  development  lifecycle. 

What  Maturity  Metrics  Do  We  Have? — Technology  Readiness  Level  (TRL) 

The  need  to  assess  the  maturity  level  of  the  technologies  and  systems  in  the 
development  process  becomes  a  critical  factor  in  the  decision-making  process  throughout  the 
system  development  lifecycle.  This  has  led  to  the  introduction  of  a  metrics  assessment  process 
supporting  the  assessment  of  maturity  of  different  types  of  technologies  used  in  a  system 
development  program.  One  of  these  metrics,  the  Technology  Readiness  Level  (TRL)  was 
originally  introduced  by  the  National  Aeronautics  and  Space  Administration  (NASA)  for  the 
development  and  support  of  their  space  mission  programs  and  later  adapted  for  use  by  other 
agencies,  including  the  DoD.  The  TRL  describes  the  maturity  level  of  that  technology.  There  are 
nine  TRL  levels  used  to  describe  the  maturity  of  a  particular  technology,  starting  from  a  TRL  1 , 
in  which  basic  principles  have  been  observed  and  reported,  and  progressing  to  a  maximum  of 
TRL  9,  in  which  the  technology  has  been  proven  in  a  successful  operational  test  (Mankins, 
1995). 
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Table  1.  NASA’s  Technology  Readiness  Levels  Summary 

What’s  New  in  Maturity  Metrics — System  Readiness  Level  (SRL) 

While  the  Technology  Readiness  Level  (TRL)  works  well  in  providing  a  common  maturity 
assessment  metric  in  system  development  involving  individual  technologies,  it  does  not  address 
those  projects  with  systems  involving  multiple  technologies.  The  introduction  and  application  of 
the  System  Readiness  Level  (SRL)  provides  a  potential  solution  to  this  problem  (Sauser, 

Verma,  Ramirez-Marquez  &  Gove,  2006).  The  SRL  metric  indicates  the  systems  maturity  level 
of  a  system  composed  of  multiple  technologies  undergoing  a  lifecycle  system  development 
effort.  It  is  a  system  maturity  index  that  can  provide  a  “snapshot”  view  of  the  system  maturity 
throughout  a  system  development  lifecycle.  The  SRL  is  formulated  by  incorporating  the 
currently  used  TRL  index  along  with  a  newly  introduced  index,  Integration  Readiness  Level 
(IRL).  The  IRL  describes  the  level  of  integration  maturity  between  any  two  system  components 
that  are  integrated.  Applying  the  IRL  methodology  for  a  particular  system  yields  a  unique  IRL 
matrix  reflecting  that  system’s  physical  architecture. 
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STEVENS 


Initltute  of  firth  nolog, y 


Integration  Readiness  Level 

A  systematic  measurement  of  the  interfacing  of  compatible  interactions  for  various  technologies  and  the 
consistent  comparison  of  the  maturity  between  integration  points. 


Integration  -  the  combining  and  coordinating  of  separate  components  into  a  seamless  unit  - 
interfacing  the  compatible  interactions  of  various  technologies  together 


IRL 

Definition 

9 

Integration  is  Mission  Proven  through  successful  mission  operations. 

8 

Actual  integration  completed  and  Mission  Qualified  through  test  and  demonstration,  in  the  system  environment. 

7 

The  integration  of  technologies  has  been  Verified  and  Validated  with  sufficient  detail  to  be  actionable. 

6 

The  Integrating  technologies  can  Accept,  Translate,  and  Structure  Information  for  its  intended  application. 

5 

There  is  sufficient  Control  between  technologies  necessary  to  establish,  manage,  and  terminate  the  integration. 

4 

There  is  sufficient  detail  In  the  Quality  and  Assurance  of  the  integration  between  technologies. 

3 

There  is  Compatibility  fi.e.  common  language)  between  technologies  to  orderly  and  efficiently  Integrate  and 
interact- 

2 

There  is  some  level  of  specificity  to  characterize  the  Interaction  (i.e.  ability  to  Influence)  between  technologies 
through  their  interface. 

1 

An  Interface  between  technologies  has  been  identified  with  sufficient  detail  to  allow  characterization  of  the 
relationship. 

Govs,  R  {2007)  Development  of  an  integration  Ontology  for  Systems  Operational  Effectiwfless,  M.S.  T  tests.  Elevens  Institute  cf  Technology.  Hdboken,  NJ 
Gove,  R.,  B.  Saussr,  J.  Ramirez-Marquez.  (2007).  "Integration  Maturity  Metncs:  Development  of  an  Integration  Readiness  Level."  International  Journal  of 
Technology  Uenegement  (tmtar  .1.1m)  0  2007  $tevens  ,ns„,ute,  of  Technology 


Table  2.  Integration  Readiness  Levels 

Though  the  SRL  concept  is  not  fully  mature  or  accepted  universally,  it  provides  the 
beginnings  of  an  effective  system  maturity  assessment  process  framework  that  can  support  and 
improve  the  decision-making  process  throughout  the  system  development  lifecycle  by  reducing 
uncertainty  and  risk.  The  SRL  metric  provides  the  following  benefits: 

■  Common  metric  methodology  that  is  easy  to  apply 

■  Integrates  well  into  system  lifecycle  framework 

■  Supports  the  management  decision-making  process. 

■  Provide  a  more  precise  “system  level”  maturity  assessment 

Calculating  the  SRL 

This  excerpt  for  Sauser,  Verma,  Ramirez-Marquez,  DiMarzio,  and  Devanandham  (2008) 
describes  the  SRL  computation  as  follows: 

The  computation  of  the  SRL  is  a  function  of  two  matrices: 

1 .  Matrix  TRL  provides  a  blueprint  of  the  state  of  the  system  with  respect  to  the 
readiness  of  its  technologies.  That  is,  TRL  is  defined  as  a  vector  with  n  entries  for 
which  the  Ith  entry  defines  the  TRL  of  the  P  technology. 
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2.  Matrix  IRL  illustrates  how  the  different  technologies  are  integrated  with  each  other 
from  a  system  perspective.  IRL  defined  as  an  nxn  matrix  for  which  the  element  IRL/, 
represents  the  maturity  of  integration  between  the  /h  and  /h  technologies. 

In  these  matrices,  the  standard  TRL  and  IRL  levels  corresponding  to  values  from  1 
through  9  should  be  normalized.  Also,  it  has  been  assumed  that  on  the  one  hand,  a  value  of  0 
for  element  IRL/,  defines  that  the  /thand /h  technologies  are  impossible  to  integrate.  On  the  other 
hand,  a  value  of  1  for  element  IRL^  can  be  understood  as  one  of  the  following  with  respect  to  the 
ithar\6  /h  technologies:  1)  completely  compatible  within  the  total  system,  2)  do  not  interfere  with 
each  others  functions,  3)  require  no  modification  of  the  individual  technologies,  and  4)  require 
no  integration  linkage  development.  Also  it  is  important  to  note  that  IRL//  may  have  a  value  lower 
than  1 ,  illustrating  that  the  technology  may  be  a  composite  of  different  sub-technologies  that  are 
not  absolutely  mature. 

In  any  system,  each  of  the  constituent  technologies  is  connected  to  a  minimum  of  one 
other  technology  through  a  bi-directional  integration.  How  each  technology  is  integrated  with 
other  technologies  is  used  to  formulate  an  equation  for  calculating  SRL  that  is  a  function  of  the 
TRL  and  IRL  values  of  the  technologies  and  the  interactions  that  form  the  system.  In  order  to 
estimate  a  value  of  SRL  from  the  TRL  and  IRL  values  we  propose  a  normalized  matrix  of  pair¬ 
wise  comparison  of  TRL  and  IRL  indices.  That  is,  for  a  system  with  n  technologies,  we  first 
formulate  a  TRL  matrix,  labeled  [TRL].  This  matrix  is  a  single  column  matrix  containing  the 
values  of  the  TRL  of  each  technology  in  the  system.  In  this  respect,  [TRL]  is  defined  in  Equation 
1 ,  where  TRL,  is  the  TRL  of  technology  /'. 


(1) 


[TRL  ],IX1 


TRLt 

TRL2 


TRL 


n 


Second,  an  IRL  matrix  is  created  as  a  symmetric  square  matrix  (of  size  nxn)  of  all 
possible  integrations  between  any  two  technologies  in  the  system.  For  a  system  with  n 
technologies,  [IRL]  is  defined  in  Equation  2,  where  IRLy  is  the  IRL  between  technologies  /'  and  j. 
It  is  important  to  note  that  whenever  two  technologies  are  not  planned  for  integration,  the  IRL 
value  assumed  for  these  specific  technologies  is  the  hypothetical  integration  of  a  technology  /'  to 
itself;  therefore,  it  is  given  the  maximum  level  of  9  and  is  denoted  by  IRL/ 


'IRLU 

IRLn  . 

..  IRL 

(2) 

§ 

!s— ' 

a 

II 

IRL2l 

IRL  22 

..  IRL 

JRLnl 

irl„2  ■ 

..  IRL 

Although  the  original  values  for  both  TRL  and  IRL  can  be  used,  the  use  of  normalized 
values  allows  a  more  accurate  comparison  when  comparing  the  use  of  competing  technologies. 
Thus,  the  values  used  in  [TRL]  and  [IRL]  are  normalized  (0,1)  from  the  original  (1,9)  levels. 
Based  on  these  two  matrices,  an  SRL  matrix  is  obtained  by  obtaining  the  product  of  the  TRL 
and  IRL  matrices,  as  shown  in  Equation  3. 

(3)  l«l.r[fliL«[wL 

The  SRL  matrix  consists  of  one  element  for  each  of  the  constituent  technologies  and 
from  an  integration  perspective,  quantifies  the  readiness  level  of  a  specific  technology  with 
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respect  to  every  other  technology  in  the  system  while  also  accounting  for  the  development  state 
of  each  technology  through  TRL.  Mathematically,  for  a  system  with  n  technologies,  [SRL]  is  as 
shown  in  DoD  (2005). 


SRLX 

~ IRL t , TRL t  +IRLl2TRL2  + ...  +  IRLuTRLn " 

[srl]= 

SRL, 

= 

1 R  L ,  |  TRL j  +  IRL22  TRL2  + . . .  +  IRL2n  TRLn 

_SRLn_ 

IRL  , TRL,  +IRL.TRL ,  +...  +  IRLTRL,, 

_  n 11  nil  nn  n  _ 

where  IRL^IRLj,. 


Each  of  the  SRL  values  obtained  in  DoD  (2005)  would  fall  within  the  interval  (0,n).  For 
consistency,  these  values  of  SRL  should  be  divided  by  “n”  to  obtain  the  normalized  value 
between  (0,1).  Notice  that  [SRL]  itself  can  be  used  as  a  decision-making  tool  since  its  elements 
provide  a  prioritization  guide  of  the  system’s  technologies  and  integrations.  Thus,  [SRL]  can 
point  out  deficiencies  in  the  maturation  process. 

The  SRL  for  the  complete  system  is  the  average  of  all  such  normalized  SRL  values,  as 
shown  in  Equation  5.  Equal  weights  are  given  to  each  technology  and  hence  a  simple  average 
is  estimated.  A  standard  deviation  can  also  be  calculated  to  indicate  the  variation  in  the  system 
maturity  and  parity  in  subsystem  development. 


rSRL  SRL,  SRL  A 
- L  + - 1  +  ...  + - ! 


(5) 


SRL  = 


ln  J 


where  ni  is  the  number  of  integrations  with  technology  /. 


The  SRL  metric  can  be  used  to  determine  the  maturity  of  a  system  and  its  status  within  a 
developmental  lifecycle. 

Applying  the  SRL  Methodology 

In  the  following  SRL  examples,  we  take  two  different  system  architectures,  each 
consisting  of  six  technologies,  and  track  the  System  Readiness  Level  metrics  through  the 
system  development  lifecycle,  calculating  the  SRL  metrics  at  each  program  decision  point.  We 
also  look  at  the  effects  of  IRL  maturity  on  the  composite  SRL  position  along  the  system  lifecycle 
by  calculating  the  SRL  for  IRLs  =  1,5,  and  9.  This  information  can  support  the  decision-making 
process  by  providing  us  with  valuable  information  about  the  maturity  of  the  system  undergoing 
development  and  the  status  of  the  system’s  individual  components.  The  two  examples  shown  in 
the  following  sections  illustrate  the  SRL  composites  mapped  across  the  entire  system  lifecycle. 
One  can  derive  some  interesting  points  by  reviewing  the  data  in  these  tables.  For  example, 
using  the  traditional  TRL  methodology  and  looking  at  Milestone  C,  we  see  that  all  the  TRLs  are 
equal  to  TRL  7.  If  we  look  at  the  SRL  composite  value  for  a  maximum  IRL  value  set  equal  to  9, 
we  see  that  the  table  data  shows  the  system  maturity  aligned  with  Milestone  C,  which  in  the 
traditional  sense  means  we  have  a  TRL  equal  to  7.  Introducing  the  new  SRL  methodology,  we 
can  show  that  for  a  TRL  7,  and  a  lower  Integration  Readiness  Level  of  IRL  5,  the  SRL 
composite  value  then  drops  the  SRL  of  the  system  to  a  point  close  to  Milestone  B.  This  point 
could  perhaps  shed  some  light  in  the  area  of  COTS  applications  where  developers  have 
assumed  their  COTS  components  to  be  at  a  high  TRL  level,  assuming  easy  and  straightforward 
integration,  and  find  themselves  with  great  difficulty  in  the  integration  process. 
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SRL  Example  1 
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Table  3.  SRL  Example  1  Mapping  to  System  Lifecycle 
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SRL  Example  2 
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Table  4.  SRL  Example  2  Mapping  to  System  Lifecycle 


Push  for  Portfolio  Management 

As  systems  become  more  complex,  management  of  their  development  efforts  become 
more  difficult.  The  management  environment  has  become  a  critical  focus  for  many 
organizations  seeking  to  ensure  the  success  of  their  programs  and  projects.  The  quest  for  new 
innovative  approaches  supporting  the  management  decision-making  process,  including  new 
software  management  tools,  are  at  the  top  of  the  list.  Portfolio  management  is  defined  as  the 
management  of  an  optimized  group  of  projects  aligned  towards  a  central  goal,  theme,  or 
strategy — sharing  common  resources  within  an  organization.  Portfolio  management  principles 
can  be  applied  on  the  corporate  level  as  well  as  the  program  or  project  level.  In  order  to 
corporate  portfolio  management  principles  to  be  effective  in  an  organization,  that  organizational 
behavior  and  process  must  be  aligned  towards  a  common  goal  or  strategy  (Sanwal,  2007). 
Though,  the  application  of  portfolio  management  strategies  to  different  domains  are  evident 
from  the  many  coined  references  like  “corporate  portfolio  management,”  “project  portfolio 
management,”  and  “enterprise  portfolio  management,”  their  basic  approaches  are  the  same. 
Recently  introduced  corporate  portfolio  management  (CPM)  ideologies  supporting  system 
development  in  the  DoD  have  shown  some  promise  in  providing  a  more  dynamic  approach  to 
project  management.  The  DoD’s  Joint  Net-Centric  Operations  (JNO)  group  has  adopted  a 
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capability  portfolio  management  process  to  ensure  that  the  portfolio  is  aligned  with  strategic 
objectives,  the  capability  mix  is  synchronized,  integrated,  and  optimized  to  meet  warfighter 
needs,  while  being  delivered  more  rapidly  and  efficiently.  The  overall  goal  of  applying  joint 
capability  portfolio  management  is  to  help  manage  groups  of  similar  and  like  capabilities  across 
the  DoD  enterprise  to  improve  interoperability,  minimize  capability  redundancies  and  gaps,  and 
maximize  capability  effectiveness  (JNO,  2007,  April). 

Developing  a  CPM  Strategy 

The  portfolio  management  process  begins  with  a  vision  or  desired  capability  that  defines 
the  strategic  focus  of  the  organization.  This  can  be  driven  by  internal  corporate  goals  and/or  by 
external  customer/stakeholder  requirements  or  needs.  These  requirements  or  needs  are  then 
translated  to  high-level,  long-term  research  development  goals  and  objectives,  which  can  then 
be  developed  and  achieved  through  a  well  defined,  executed  program.  The  final  deliverable  to 
the  customer  will  be  a  technological  capability,  which  is  delivered  to  the  customer  through  a 
technology  transfer  process.  These  high  level  principles  are  highlighted  in  a  recent  INCOSE 
paper  titled,  “A  Systems  Approach  to  the  Transition  of  Emergent  Technologies  into  Operational 
Systems — Herding  the  Cats,  the  Road  to  Euphoria  and  Planning  for  Success,”  which  discusses 
the  critical  elements  needed  to  support  and  enable  successful  technology  transition  through  the 
lifecycle  development  process  (Austin,  Zakar,  York,  Pettersen  &  Duff,  2008). 

Four  Key  Questions  Driving  CPM  Strategy 

1.  What  are  we  trying  to  Accomplish?  (Euphoria) 

This  question  asks  “’’Where  do  you  want  to  be?”  and  drives  an  end-state  vision  and  goal 
based  on  high-level  corporate  strategy  and  stakeholder  requirements. 

2.  What  can  we  do  now?  (Herding  the  Cats) 

Here,  we  must  determine  “Where  are  we  now?,”  “What  can  we  do  now?,”  “What  are  our 
technical  assets,  past  accomplishments,  and  available  resources?”  and  “Can  they  be  aligned 
with  the  desired  end-state  goals? 

3.  What  is  our  plan  to  get  there?  (the  Road  to  Euphoria) 

Based  on  the  answers  to  the  first  two  questions,  identify  the  technology  gaps,  and 
develop  a  roadmap  or  plan  to  reach  the  desired  goals. 

4.  How  are  we  doing?  (the  Metrics) 

Here,  we  need  to  determine  how  well  the  system  lifecycle  development  is  maturing  so 
that  corrections  and  modifications  can  be  implemented  if  necessary. 
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Figure  5.  Lifecycle  Corporate  Portfolio  Management 
Portfolio  Enterprise  View 

Based  on  the  answers  to  the  “Four  Key  Questions  Driving  CPM  Strategy”  discussed  in 
the  previous  section,  the  selection  of  projects  is  based  on  their  alignment  to  the  desired 
capabilities  and  sub-objectives  as  weil  as  available  resources,  including  funding  and  available 
manpower. 

Implementation  of  the  portfolio  management  approach  to  project  management 
eliminates  the  traditional  approaches  that  led  to  multiple  concurrent,  often  duplicative  and 
“stove-piped”  solutions  that  were  inefficient,  often  subjected  to  irrational,  “below  the  line”  and 
“salami  slice”  budget  cuts.  These  cuts  can  result  in  key  capabilities  being  lost,  leading  to 
programs  not  being  able  to  meet  their  objectives.  Portfolio  management  presents  an 
“enterprise”  approach,  providing  for  synchronized  investments  to  deliver  maximum  capability 
through  the  prioritization  of  your  investments  by  maintaining  an  optimal  mix  of  investments  in 
objectives  aligned  to  your  strategy. 
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Figure  6.  Portfolio  Project  Enterprise  View 


Non-Enterprise  Approach: 

Multiple  concurrent,  stovepiped 
projects  without  consistent  focus 
reduces  effectiveness  of 
capability 
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Enterprise  Approach 

Analysis  of  all  projects  with  future 
objectives  reduces  redundancy  and 
increases  capability 
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Figure  7.  Historical/Enterprise  Approaches 


Lifecycle  Portfolio  Management 

Lifecycle  portfolio  management  includes  the  capture  of  a  variety  of  metrics  (financial, 
performance,  and  maturity  metrics)  that  are  analyzed  and  the  results  support  some  type  of 
decision-making  process.  At  this  point,  optimization  is  considered  a  way  to  keep  the  portfolio 
better  aligned  to  meet  it  strategic  goals. 
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Metrics  Assessments 

The  strength  of  portfolio  management  lies  in  the  capture  of  metrics  that  measure  the  vital 
functions  of  a  development  effort  and  how  well  the  development  process  is  going.  These 
metrics  provide  input  to  the  optimization  and  decision-making  process.  These  metrics  can  be 
captured  on  a  quarterly  basis  and/or  tied  to  key,  program-specific  development  milestones. 
Progress  against  these  milestones  can  provide  key  insight  to  the  user  regarding  current 
program  status,  risk  and  progress.  After  the  initial  strategy  development  phase,  a  proposed 
approach  for  applying  the  maturity  metrics  to  the  portfolio  management  process  would  include 
performing  the  following: 

■  Initial  assessment  of  selected  technologies  in  portfolio  mix,  which  includes  the  initial 
assessment  TRL/IRL/SRL  data,  resource  data. 

■  Quarterly  cycle  assessments  of  TRL/IRL,  SRL,  and  funding  and  at  milestones  A,  B, 

C  of  the  DoD  5000.2  system  lifecycle  framework. 

■  Ongoing,  search  for  new  and  viable  technologies  that  may  be  available  now  or  in  the 
near  future  for  possible  integration  or  substitution  into  existing  portfolio  mix. 

■  Analysis  of  data  to  see  if  optimization  opportunity  is  available. 


Life  Cycle  Portfolio  Management  Activities 
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Figure  8.  CPM  Lifecycle  Activities 

Optimization 

One  of  the  key  focuses  of  successful  portfolio  management  is  trying  to  maintain  an 
optimal  mix  of  technology  development  efforts  aligned  with  the  organizations  strategic  vision  or 
goals.  In  addition,  we  need  to  understand  how  well  or  how  fast  these  individual  technologies  are 
maturing  relative  to  each  other  or  if  any  new  external  technologies  have  been  developed  and 
can  be  immediately  substituted,  allowing  for  more  dynamic  changes  to  the  portfolio  mix.  How  do 
you  decide  between  competing  system  design  alternatives  or  which  individual  TRL  or  IRL  to 
improve?  The  use  of  optimization  modeling  techniques  can  provide  great  insight  and  support  to 
trade-off  analysis  and  decision-making  throughout  the  system  development  lifecycle. 
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Two  recently  developed  optimization  models,  System  Cost  of  Development  (SCOD) 
Minimization  and  SRL  Maximization  are  examples  of  using  optimization  techniques  to  help 
provide  better  decision-making,  control-based  resource  constraints.  The  first  model,  SCOD 
Minimization,  considers  minimizing  the  development  cost  associated  to  increasing  SRL  to 
some  predefined  user  level,  A.  This  model’s  objective  is  to  minimize  development  cost  (a 
function  of  TRL  and  IRL  development)  under  constraints  associated  with  schedule  and  the 
required  SRL  value  (Magnaye,  Sauser,  Ramirez-Marquez  &  Tan,  2008).  The  second  model, 
SRL  Maximization,  maximizes  the  SRL  (a  function  of  TRL  and  IRL)  under  constraints 
associated  with  resources.  This  model  recognizes  that  the  technologies  compete  for  resources 
and  that  benefits  can  result  in  an  improved  SRL  via  the  optimal  allocation  of  such  resources 
(Sauser  &  Ramirez-Marquez,  2009).  In  summary,  optimization  modeling  should  help  provide  the 
decision-maker,  whether  it  is  the  program  manager  or  the  systems  engineer  with  the  best 
balance  between  the  SRL  and  all  the  associated  resources  to  help  achieve  the  desired  end- 
state  goals.  We  must  remember  that  optimization  should  be  considered  only  a  tool  used  along 
with  other  inputs,  like  metrics,  to  help  provide  depth  to  the  decision-making  process. 

Decision-making 

CPM  decision-making  is  a  complex  undertaking  as  there  are  many  elements  and  events 
that  need  to  be  understood  and  analyzed  in  a  real-time  manner.  The  pressures  of  schedule, 
cost  and  performance  hold  true  along  with  an  associated  more  real-time  element.  Adherence  to 
the  DoD  5000  acquisition  framework’s  critical  decision  point  assessments  at  milestones  A,  B, 
and  C  affects  the  optimization  process. 

1.  Optimal  mix  of  research  development  investments  to  achieve  capability  goals 
based  on  maturation,  cost,  etc. 

2.  Allocation  of  resources  to  investments  (Funding/Manpower) 

3.  Corrections  to  mix  of  research  investments  in  reaction  to  the  introduction  of 
new  technologies 
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Figure  9.  Portfolio  Management  Decision  Engine 

Summary  and  Conclusion 

The  purpose  of  this  paper  was  to  introduce  the  concepts  of  SRL  metrics  to  multi¬ 
technology  based  system  development  environment  in  a  portfolio  management  environment. 
The  proposed  application  of  these  concepts  and  ideologies  presents  a  new,  potentially  viable 
alternative  to  previously  methodologies  using  TRL  metrics.  Stressed  in  this  paper  was  the  belief 
that  you  must  consider  an  integrated  approach  to  ensure  that  the  portfolio  management  process 
and  system  maturity  metric  assessment  process  are  synchronized  closely  to  a  lifecycle 
framework  in  order  to  meet  your  strategic  goals.  Looking  ahead,  research  in  the  following  areas 
would  further  contribute  to  the  body  of  knowledge  in  System  Maturity  Metrics: 

■  SRL  software  tools  to  implement  a  combined  SE,  CPM  and  Road  Mapping. 

■  Application  of  SRL  metrics  to  support  CPM  environment. 

■  What  additional  maturity  metric  variables  are  needed  to  support  the  decision-making 
process? — security  readiness 

■  Application  of  SRL  model  to  other  lifecycles  outside  the  DoD. 

■  Robustness  of  SRL  to  variety  of  differing  physical  architectures. 

■  Impacts  of  disruptive  technologies  on  systems  maturity  forecasting. 

■  SRL  applications  to  COTS  environment  and  lifecycle  development 
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Introduction 

■  Increasing  number  of  major  DOD  system 
development  programs  experiencing 
difficulties  and  failing  to  achieve  their  intended 
goals  successfully. 


■  Resulting  in: 

-  Cost  overruns 

-  Program  delays 

-  Program  cancellations 

-  Unacceptable  system  performance. 
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System  Development  Challenges 


■  Systems  have  become  far  more  complex 

■  Increased  data  demand  requirements 

■  Operating  in  a  net-centric  environment 

■  Increasing  threats  to  system  security 

■  Rapid  development  cycle 

■  Rapid  technology  obsolescence 

■  Funding  constraints 

■  Experienced  workers. 
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System  Development  Challenges  -  “S”Curves 
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System  Development  Challenges 


■  According  to  various  GAO  studies  of  DOD  technology 
development  practices,  reasons  for  these  difficulties  are  the 
inability  to  assess  technical  maturity  of  complex  systems 
during  development 

■  1999  -  GAO  report  reviewing  major  defense  acquisition 
programs  and  analyzing  the  causes  and  reasons  for  a 
majority  of  them  and  their  failure  to  meet  at  least  a  TRL  7 
level  before  entering  the  system  development  phase. 

■  2008  -  GAO  report  showed  an  increase  from  the  previous 
year  in  the  number  of  programs  with  immature  technologies 
still  maturing  technologies  late  into  the  system  development 
and  production  live  cycles.  (9  yrs  after  similar  report) 

■  2007  -  DoD  Report  to  Congress  -  Need  to  Establish  a 
Process  to  Enable  a  “Systematic  Approach  to  Product 
Development” 
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What  are  Maturity  Metrics? 

■  What  are  Maturity  Metrics?  -  Metrics  supporting  the 
lifecycle  assessment  of  a  system  or  technology’s  state 
of  progress  or  development. 

■  We  have  made  considerable  improvements  in  the  area 
of  improved  software  IT  systems  to  perform  financial 
status  tracking  and  monitoring  metrics  of  system 
development. 


■  Importance?  -  Assessment  of  the  maturity  level  of  the 
systems  and  technologies  are  a  critical  factor  in  the 
decision  making  process  throughout  the  system 
development  lifecycle. 
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What  do  we  have  now? 


-  Technology  Readiness  Levels  (TRLs) 


■  Describes  the  maturity  level  of 
a  technology  (9  levels) 

■  Introduced  by  NASA  for  their 
space  programs 

■  Later  adapted  for  use  by  other 
agencies  (DoD) 

■  Supports  the  maturity 
assessment  of  individual 
technologies  well 

■  Doesn’t  address  assessment 
of  systems  involving  multiple 
technologies. 


TECHNOLOGY  READINESS  LEVELS  (TRL’s) 
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operations 

Actual  system  completed  and  qualified  through  tost 
and  demonstration 

System  prototype  demonstration  in  a  re  [event 
environment 

System/s  Lib  system  model  or  prototype  demonstration 
in  a  relevant  environment 

Component  and/or  breadboard  validation  in  relevant 
environment 

Component  anchor  breadboard  validation  in  laboratory 
environment 

Analytical  and  experimental;  critical!  function  and/or 
characteristic  proof-of-concapt 

Technology  concept  and/or  application  formulated 


Basic  principles  observed  and  reported 
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What’s  New  in  Maturity  Metrics 

-  System  Readiness  Level  (SRL) 

■  Describes  the  maturity  level  of  a  system  comprised  of 
multiple  technologies  (9  levels) 

■  Proposed  by  Stevens  Institute  of  Technology  to  address 
need  for  system  maturity  metrics  for  multi-technology  based 
system  development  not  address  by  current  TRL  metrics 


■  SRL  Model  -  Incorporated  currently  used  TRL  index  with 
new  index,  Integration  Readiness  Level  (IRL). 


■  IRL  describes  how  the  system  components  are  integrated 
together,  (related  to  physical  architecture  of  system) 
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What’s  New  in  Maturity  Metrics 
-Integration  Readiness  Levels 


STEVENS 


InsiitutcolicdinciEo^' 


Seiuduf 


Integration  Readiness  Level 

A  systematic  measurement  of  the  interfacing  of  compatible  interactions  for  various  technologies  and  the 
consistent  comparison  of  the  maturity  between  integration  points , 


Integration  -  the  combining  and  coordinating  of  separate  components  into  a  seamless  unit  ■ 
interfacing  the  compatible  interactions  of  various  technologies  together 
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Integration  is  Mission  Proven  through  successful  mission  operations. 

8 

Actual  integration  completed  and  Mission  Qualified  through  test  and  demonstration.  In  the  system  environment. 

7 

The  Integration  of  technologies  has  been  Verified  and  Validated  with  sufficient  detail  to  be  actionable. 

6 

The  integrating  technologies  can  Accept,  Transfate,  and  Structure  Information  for  its  intended  application. 

5 

There  is  sufficient  Control  between  technologies  necessary  to  establish,  manage,  and  terminate  the  integration. 

4 

There  is  sufficient  detail  in  the  Quality  and  Assurance  of  the  integration  between  technologies. 

3 

There  is  Compatibility  [i,e.  common  language)  between  technologies  to  orderly  and  efficiently  Integrate  and 
interact- 

2 

There  is  some  level  of  specificity  to  characterize  the  Interaction  {Le.  ability  to  Influence)  between  technologies 
through  their  interface- 

1 

An  Interface  between  technologies  has  been  identified  with  sufficient  detail  to  allow  characterization  of  the 
relationship. 

Gove,  R.  (200?)  Devstopment  of  an  jfntegrairor?  Onfafogy  for  Systems  Operational  Effedhvn-ess  M.S.  Thesis.  Stevens  institute  of  Technology.  Hoboken,  NJ 
Gove„  R..  S.  Sauser,  J.  Ramiraz-Manpiez.  (2007)  “Integration  Maturity  M&lncs:  Development  of  an  Integration  Readiness  Level '  intematiomi  Joomut  of 
T^ndogy  Uwgm, «* lurrf*  nta.)  0  2007  $teven,  1n5tilgte  of  Technology 
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Applying  the  SRL  -  Example  1 
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Applying  the  SRL  -  Example  2 
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Push  for  Portfolio  Management 


■  DoD:  Joint  Net-Centric  Operations  (JNO)  group  adopted  a 
capability  portfolio  management  process  to  ensure  that  the 
portfolio  is  aligned  with  strategic  objectives,  and  the  capability 
mix  synchronized,  integrated,  and  optimized  to  meet  warfighter 
needs,  rapidly  and  efficiently.  (JNO.  2007,  April). 

■  CPM  Highlights: 

■  Ideal  for  large  programs  (multiple  projects) 

■  Focuses  on  Project  Selection,  Prioritization,  Resource 
Allocation,  Strengths/Weakness  of  each  project 

■  Identifies  Gaps/future  development  opportunities 

■  Determines/manages  optimal  mix  of  development 
projects  to  achieve  capability  goals  and  objectives 
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Developing  a  CPM  Strategy 


Four  Key  Questions: 

1.  What  are  we  Trying  to  Accomplish? 

(Euphoria) 

2.  What  can  we  do  now? 

(Herd  the  Cats) 

3.  What  is  our  Plan  to  get  There? 

(Road  to  Euphoria) 

4.  How  are  we  Doing? 

(Metrics) 
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Developing  a  CPM  Strategy 
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Developing  a  CPM  Strategy  -  Enterprise  View 
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Developing  a  CPM  Strategy  -  Approaches 


Non-Enterprise  Approach: 

Multiple  concurrent,  stove-piped  projects 
without  consistent  focus  reduces 
effectiveness  of  capability 


Enterprise  Approach: 

Analysis  of  all  projects  with  future 
objectives  reduces  redundancy  and 
increases  capability 


Service  Project , 
Agency  Project 


Service  Project  B 


Service  Project  D 
Agency  Project  G 

V 


Service  Project  C 


Agency  Project  F 


Service  Project  X 
Service  Project  B-C 
lAgency  Project  F 


Recommendations: 

►  Keep  Agency  Project  F 

►  Combine  Service  Project  B&C 

►  Add  new  Service  Project  X 

►  Reallocate  $20M  savings  to  other  investments 

►  Disinvest  in  redundant  Projects  (A,D,E  and  G) 
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Developing  a  CPM  Strategy  -  Approaches 


Service/Agency  Historical  Approach 
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Developing  a  CPM  Strategy 

Lifecycle  CPM  Metrics 
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Developing  a  CPM  Strategy 

Optimization  Models 

Provide  great  insight  and  support  to  trade-off  analysis  and 
decision  making  throughout  the  system  development 
lifecycle. 

■  SCOP  Min  -  Minimizes  development  cost  (a  function 
of  TRL  and  IRL  development)  to  some  predefined  user 
level,  A,  under  constraints  associated  with  schedule  and 
required  SRL  value  (Magnaye,  Ramirez-Marquez,  & 
Tan,  2008). 


■  SRL  Max  -  Maximizes  the  SRL  (a  function  of  TRL  and 
IRL)  under  constraints  associated  with  optimal  allocation 
of  resources.  (Sauser  &  Ramirez-Marquez,  2009). 
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Developing  a  CPM  Strategy 


Decision  Making  -  Complex  due  to  many 
elements  and  events  that  need  to  be  understood, 
analyzed,  in  a  real-time  manner. 

■  Pressures  of  schedule,  cost  and  performance 
still  hold  true  with  added  real-time  element. 


■  Allocation  of  Resources  to  investments 
(Funding/Manpower) 


■  Corrections  to  mix  of  research  investments  in 
reaction  to  introduction  of  new  technologies 

■  Optimal  mix  of  research  development 
investments  to  achieve  capability  goals 


Acquisition  Research  Program;  Creating  Synergy  for  Informed  Change 


llUSiftr ill) LUlIc  V 111 Hhl 
Mtmlcrey,  <l\ 


23 


Developing  a  CPM  Strategy 

Decision  Making 
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Summary  and  Conclusion 

Introduction  of  the  following: 

■  Integrated  approach  to  ensure  the  CPM 
process  and  system  maturity  assessment 
process  are  synchronized  to  a  lifecycle 
framework. 

■  Application  of  a  SRL  methodology  to  multi¬ 
technology  based  system  development  in  a 
CPM  environment 

■  CPM  strategy  and  decision  making  process 


Summary  and  Conclusion 

Future  Research 

•  System  maturity  metrics  to  benefit  and  improve  performance  of 
existing  DOD  system  development  programs. 

•  Application  of  SRL  metrics  to  support  CPM  environment. 

•  Development  of  integrated  S/W  tools  to  support  SE,  CPM  and 
Road  Mapping  capabilities. 

•  Identification  of  additional  maturity  metric  variables  needed  to 
support  the  decision  making  process? 

•  Application  of  SRL  model  to  other  life  cycles  outside  DoD. 

•  Robustness  of  SRL  model  to  variety  of  differing  physical 
architectures. 


Impacts  of  disruptive  technologies  on  systems  maturity 
forecasting. 

SRL  applications  to  COTS  environment  and  lifecycle 
development 

Addition  of  other  variables  to  SRL  model  -  security  readiness 
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